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ABSTRACT 



This study compared college students who used the World Wide 
Web versus paper and pencil to respond to one of three surveys: the 1997 and 
1999 Student Experiences Survey and 1998 University of Minnesota Graduate 
Survey. Students chose to respond by mail or via the Web. Researchers 
examined the proportion of students who used the Web versus those using paper 
and pencil; whether students using the Web differed significantly from those 
responding on paper; whether attitudes, experiences, and satisfaction levels 
of Web respondents differed significantly from those responding on paper; and 
whether rates of Web choice increased over time as respondents became more 
familiar with the Internet. Most students preferred paper and pencil, 
although the proportion of students selecting the Web doubled over time, 
suggesting that as students become more experienced in using the Web, their 
use increases. Younger students, who were more familiar with the Internet, 
selected the Web-based survey more often than older students. Men were more 
likely than women to use the Web-based survey. Current students in 1999 who 
selected the Web were significantly more likely to give the university high 
ratings in academic quality than were students who responded on paper. (SM) 
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A World Wide Web Response to Student Satisfaction Surveys: 
Comparisons Using Paper and Internet Formats 



Abstract 

More institutional researchers view the World Wide Web as a vehicle to gather and 
display information, but initiatives to expand survey research to the Web have had mixed 
results. This investigation compares students who used a Web interface with those who used a 
traditional paper-and-pencil format to respond to one of three surveys. The authors compare 
students’ responses, response rates and demographic characteristics by their method of 
response. Students responded more frequently to multi-paged surveys using paper-and-pencil 
formats. Men were more likely than women to respond to the Web format; increases in Web 
responses were highest for freshmen and sophomores. Percentages of students who responded 
using the Web doubled over a two-year period. Data suggest that as students gain comfort 
with the Web, responses to Web-based surveys will increase. 
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A World Wide Web Response to Student Satisfaction Surveys 

A World Wide Web Response to Student Satisfaction Surveys: 
Comparisons Using Paper and Internet Formats 

A review of topics presented at the Association of Institutional Research (AIR) forums 
suggests that discussions on institutional methods and the delivery of survey research have 
increased in frequency, and continue to remain topics of discussion. This increase is 
especially true for surveys that include questions about the personal opinions and attitudes of 
current students and recent graduates, faculty, employees and their external constituencies. 
Yet, creating a well-designed survey with a high rate of response is often a challenge, even for 
the most seasoned researcher. One of the difficulties of institutional research arises in how to 
cover the costs of creating well-designed surveys, which are often expensive and time- 
intensive. 

In the last three years, use of the Internet has expanded dramatically. Recent sales in 
on-line retail and participation rates in on-line banking point to a society that is more 
accepting of technology in the modem world. Thus, as the popularity of the Internet increases 
among college students, uses of World Wide Web surveys become more salient as a way to 
reduce postage costs and to shorten the time lag between mailing a survey and having the 
information in hand for decision-making purposes. 

Initiatives to expand paper surveys to computer-administered formats have been 
welcomed with mixed results. Studies demonstrate that electronic surveying can reduce 
mailing costs and shorten the turn-around times for sending and receiving information 
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A World Wide Web Response to Student Satisfaction Surveys 
(Kiesler & Sproull, 1986; Parker, 1992.) In one study, cadets at the Air Force Academy 
responded more readily to a computerized survey than to a paper-and-pencil form, with no 
significant differences in their patterns of responding when paper and Web responses were 
compared (Antons, Dilla & Fultz, 1997.) Yet, in a study of a listserv discussion group, 
repeated electronic solicitations to complete an e-mail survey resulted in a response rate of 
only 24 percent (Meehan, & Burns, 1997.) 

Research suggests that students’ response rates to Web-based and other electronic 
survey formats can vary depending on their level of comfort with instructions, familiarity with 
the Internet, ease in completing the survey, need for confidentiality and/or personal 
characteristics such as gender, age, educational level and family income (Schlough & 
Bhuripanyo, 1998; Coates & Fansher, 1997; Saunders, Loren & Maylone, 1998.) As with 
paper-and-pencil surveys, differences in response outcomes point to the importance of 
research methods in influencing response rates (Gaddis, 1998.) Yet, as the use of the Internet 
becomes more common in colleges and universities, students may be more likely to respond to 
Web-based surveys with each new entering academic class. 

Purpose 

This study compares the responses, response rates and demographic characteristics of 
college students who replied to questions on a 1997 and 1999 Student Experiences Survey and 
to a 1998 U of M Graduate Survey. In all three studies, students were offered a choice between 
responding by mail or via the Web. 
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A World Wide Web Response to Student Satisfaction Surveys 

Specific questions the study addressed included: 

1 ) What proportion of current students and graduates will choose to respond to a Web 
survey rather then a paper-and-pencil format? Does the response rate to these 
surveys vary depending on the type of survey used? 

2) Do the students who respond to Web surveys differ significantly from those who 
choose to respond using paper-and-pencil formats on such characteristics as 
gender, ethnicity or age? 

3) Do the attitudes, experiences or satisfaction levels of students who respond via the 
Web differ significantly from those who respond by paper-and-pencil? 

4) As students become more familiar with the Internet, do their response rates to 
Web surveys increase over time? 

Method 

Sample 

The study examined the responses from the students who replied to a Student 
Experiences Survey in 1997 and 1999, along with a comparable group of recent graduates who 
replied to a U of M Graduate Survey in 1998. Data were gathered for the 1997 and 1999 
surveys using a random sample of currently enrolled undergraduates from the Twin Cities’ 
Campus. Sample 1 consisted of 866 undergraduate students (55.7% of a random sample of 
1591 students) who replied to the 1997 Student Experiences Survey using either a Web 
interface or paper and pencil format. Sample 2 consisted of a comparable group of 2, 626 
undergraduate students who replied to a 1999 Student Experiences Survey, a slightly modified 
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A World Wide Web Response to Student Satisfaction Surveys 
version of the 1997 Student Experiences Survey. Significantly more students were selected 
for the second sample to allow for greater comparisons across data categories. 

Sample 3 included 769 students (56.5% of a random sample of 1360 students) who 
responded to a survey in 1998. The sample used data from Spring 1996 degree recipients 
from the University of Minnesota, Twin Cities. 

Procedure 

The 1997 and 1999 Student Experiences Surveys were eight-page surveys developed 
by a committee of institutional researchers to assess students’ experiences and levels of 
satisfaction on a variety of indicators including the quality of teaching, quality of advising, 
experiences with computers, and general satisfaction with the student experience. Surveys 
were sent to a random sample of currently enrolled undergraduates in the spring of each year 
to assess their levels of student satisfaction within a large research institution. In 1998, an 
eight-page, U of M Graduate Survey was mailed to persons who graduated from the same 
university in 1996 to survey their views of the university and their post-graduation 
experiences. All surveys were available for students to respond by mail or by the Web. To 
enter the web survey, students had to enter a code number that was enclosed in the mailing of 
the questionnaire. 

The Web-based surveys were programmed using Cold Fusion software, and data were 
stored in an Access database. Students were contacted by mail and were sent a mail survey 
with the option of responding via the Web by accessing the Web address and by typing in their 
name, password and survey ID. Students who failed to respond to each of the surveys 
received a series of reminders. Postcards, follow-up reminders and multiple mailings 
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A World Wide Web Response to Student Satisfaction Surveys 
of the surveys were sent to those who failed to respond within a two- week interval. Prize 
incentives were also used to increase the response rates. After a series of multiple mailings, 
response rates ranged from 56 percent to 60 percent for each of the surveys. Descriptive 
statistics and a series of chi-squared tests were used to compare the characteristics, 
experiences and attitudes of students to each survey by the format used, given the unequal 
sample sizes and nonnormal distributions. For the process of the chi-squared test, items which 
used a 5 or 6 point rating scale were recoded , so that responses from fair to very poor were 
coded as ‘low’ and responses from good to excellent were coded as ‘high’. 

Results 

Responses to the Web and Paper Surveys by Survey Type and Format 

Response rates for each of the surveys are provided in Table 1 and are based on the 
proportion of students who responded to the surveys using the Web or paper-and-pencil 
formats. When given the option, students were significantly more likely to respond to the 
multi-paged surveys using the paper-and-pencil format. Web-based responses to the Student 
Experiences survey doubled over time, increasing from 7.4% in 1997 to 14.9% in 1999. 
Differences were statistically significant (p<.01.) 

Respondents to the 1998 U of M Graduate Survey were about 2 percent more likely to 
respond using the Web-based format than were students who responded to the 1997 Student 
Experiences Survey. However, the graduates were almost 6 percent less likely to respond to 
the Web-based format than those who responded to the 1999 survey. 
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Table 1 : Response Rates of Students, by Survey Type and Response Format 







Response Format 


Survey Type 
Year 


Total 

Respondents 


Web 

% 


Paper-and- Pencil 

% 


Student Experience 
Survey 








1997 


866 


7.4% 


92.6% 


1999 


2627 


14. 9% 


85. 1 % 


Graduate Survey 








1998 


796 


9.2% 


90.8% 



Response Rates to Web Surveys bv Gender. Ethnicity. Class and Collegiate Unit 

Table 2 presents the response rates for the three Web-based surveys by gender, 
ethnicity, class and collegiate unit. Responses to the Web-based formats were analyzed on the 
basis of gender, ethnicity (coded as white and non-white/other), class and collegiate unit. Men 
were significantly more likely than women to respond to the Web-based format in each 
year, when chi-square statistical tests of significance were used. Over time, the proportions of 
students who responded by the Web doubled in frequency for each gender. 
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No significant differences in response formats emerged between the White and Non- 
White/Other comparisons in ethnicity for those who responded to the Student Experiences 
Survey. Yet, some caution should be used in interpreting the data for 1999, since many 
students chose not to respond to this item. However, international students, included in the 
“Other” category did respond in higher proportions to the Web than did the White majority. 
Some statistically significant differences did appear for respondents of the Graduate Survey 
(p <.04), with White students responding more readily to a Web-based survey than nonwhite 
students. 

Web response rates differed by class level for two of the three survey years. 

No explanatory trends in students’ responses to the Web-based survey emerged by 
undergraduate class level for respondents in 1997. Yet, by 1999, lower division students were 
significantly more likely to respond to a Web-based survey than were upper division students. 
In addition, undergraduates who attained their degrees in 1996 were statistically more likely to 
respond to a Web-based version of the Graduate Survey in 1998 than were students who 
attained a graduate degree during the same year. Thus, younger students were more likely to 
respond to a Web survey than were older students. 

Statistically significant differences in collegiate unit were apparent in all of the 
surveys. Students enrolled in the area of Technology were significantly more 
likely respond to the survey using the Web format than were students from other disciplines. 
However, the proportions of students who responded using the Web option increased over 
time for all students in all disciplines, except for those enrolled in education. 
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Table 2. Percentages Who Responded to a Web Survey by Demographic Characteristics 
and Survey Type 



Variable 


Student Experiences Survey 
1997 1999 


Graduate Survey 
1998 


n 


% Web 
response 


n 


% Web 
response 


n 


% Web 
response 


Gender 














Male 


378 


11.6 


1165 


22.7 


332 


14.2 


Female 


482 


4.8 


1482 


8.8 


437 


5.5 


Ethnic Status 














Non-White 


153 


9.2 


169 


14.4* 


105 


6.0 


White 


700 


7.6 


257 


17.2 


661 


9.7 


Undergraduate Level 












Freshman 


160 


11.3 


204 


17.2 


not applicable 


Sophomore 


170 


5.9 


551 


20.3 


not applicable 


Junior 


184 


4.9 


450 


14.9 


not applicable 


Senior 


199 


10.6 


1041 


12.6 


not applicable 


Class Level 














Undergraduate 


not available 


not available 


653 


10.0 


Graduate/Prof 


not available 


not available 


103 


2.9 


College 














Business 


29 


6.9 


67 


11.9 


61 


11.5 


Education 


22 


4.5 


27 


3.7 


26 


0.0 


Technology 


163 


15.3 


584 


24.5 


136 


16.2 


Bio Sci 


48 


6.3 


293 


13.3 


29 


20.7 


Lib Arts 


357 


7.9 


742 


14.3 


357 


7.6 



*83% did not respond to the Web survey. 
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A separate analysis, which is not shown in the table, compared the income level of 
students who responded to the Web survey with persons who responded to a paper-and-pencil 
option, based on their response to a question which asked them to provide information on 
their personal, pre-taxable income for 1997. Nine possible salary categories were provided 
ranging from ‘under $10,000’ to ‘$80,000 or more.” In this comparison, Web response 
formats for persons with incomes above $60,000 were compared to those with incomes below 
this amount. No significant differences emerged between response format and income level 
for the graduates. 

Response Rates to Web Surveys by Computer Experiences 

All of the surveys contained questions relating to students’ access to and use of 
computer technology. (Data are presented in Table 3.) A total of eight items were asked 
across the three surveys, but the questions were not uniform from survey to survey. The data 
in Table 3 are only presented for the surveys in which they appear. 

In all surveys, current students and graduates who had accessed information on the 
Web were significantly more likely to respond to a survey using the Web. In 1997 and 1999, 
having 24 hour access to computers was related significantly to increasing ones’ likelihood of 
responding to a Web-based survey format. By 1999, this likelihood of responding to a Web 
survey correlated significantly with such experiences as having received or turned in an 
assignment electronically in class. 

In contrast, experiences with a word processing package or prior experiences in 
accessing on-line library services were not related to ones’ likelihood of responding to a Web- 
based survey. Findings for those who had taken courses delivered through the Web were 
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A World Wide Web Response to Student Satisfaction Surveys 

Table 3. Percentages of Students Who Responded to a Web-Based Survey As Related 
To Their Experiences With Computers and Technology 



Student Experiences Survey 

1997 1999 



Graduate Survey 
1998 



Survey Item Percent Percent Percent Percent Percent Percent 





n Yes 


n 


No 


n 


Yes 


n 


No 


n 


Yes 


n 


No 


24 Hr Access 
Computers 


561 


11.1 


301 


1.7* 


1629 


17.4 


923 


10.4* 


N/A 


N/A 


N/A 


N/A 


Web Course 


514 


8.9 


348 


6.0* 


2255 


15.0 


314 


12.2 


10 


1.4 


749 


9.2* 


Sent/Received 

Email 


809 


8.3 


0 


0.0* 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Turned In/ 
Received 


N/A 


N/A 


N/A 


N/A 


1532 


17.3 


1033 


11.4* 


N/A 


N/A 


N/A 


N/A 



Assign. 

Electronically. 



Accessed 



Web 

Information 


774 


8.5 


89 


1.1* 


1383 


16.8 


1191 


12.9* 


394 


13.7 


369 4.1* 


Accessed 

Lumina 


740 


7.7 


123 


8.1 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Accessed 
Public Lab 


732 


8.2 


130 


5.2 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Used Word 
Process Pkg. 


788 


7.9 


75 


6.7 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 



*p< .05 
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A World Wide Web Response to Student Satisfaction 
on the Internet , thereby encouraging them to respond to questionnaires, exams or other 
on-line tasks. 

Ratings of Satisfaction by Those Responding to Web Versus Paper and Pencil Formats 

This section provides information on students’ ratings of satisfaction, attitudes towards 
University services and experiences with the University, based on their responses to 
approximately seventy different items contained within three eight-page surveys. Only one 
significant difference emerged on global items relating to academic quality, overall 
satisfaction and to satisfaction with the quality of teaching or advising. Current students in 
1999 who responded to the Web format were significantly more likely to give the University 
high ratings in academic quality. In addition, University graduates who responded using the 
Web rated the University significantly higher on items relating to their employment following 
graduation and to the preparation they felt they had gained to perform in their careers. 
Percentages of ratings for Web responses appear in Table 4. 

Students who responded to the Graduate Survey rated the University on over twenty 
different skill areas. Those who responded through the Web rated the University significantly 
higher in relation to the computer skills they had gained and in their ability to locate 
information. Graduates gave slightly higher ratings to items associated with networking, 
adaptation to change, creativity, mastery of job skills and persistence, though most findings 
were not statistically different. A sample of the ratings for items analyzed appear in Table 5. 
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Table 4. Overall Satisfaction of People Who Responded to A Survey Via the Web 



Student Experiences Survey Graduate Survey 



1997 1999 1998 

High Low High Low High Low 



Survey 

Item 


n 


% 


n 


% 


n 


% 


n 


% 


n 


% 


n 


% 


Academic 

Quality 


681 


7.3 


174 


9.8 


2240 


15.8 


334 


10.8* 


N/A 


N/A 


N/A 


N/A 


Overall 

Satisfaction 


712 


8.3 


144 


4.9 


2248 


14.7 


356 


16.3 


688 


9.4 


7.9 


7.6 


Instruction 


662 


8.3 


204 


5.9 


1989 


15.4 


612 


13.4 


669 


9.0 


99 


11.1 


Advising 


302 


8.9 


481 


6.7 


1065 


14.9 


1338 


14.8 


381 


10.0 


385 


8.6 


Sat.Employ 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


484 


10.0 


98 


3.1* 


Prep.Caree 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


362 


11.6 


395 


7.3* 



r 



*p<.05 



The responses for students who answered a survey through the Web appear in 
Table 6 for nine of the comparisons. Students in 1997 who responded by the Web format 
rated the University significantly lower in a variety of areas including cost, response to 
concerns and places available to study. In contrast, in the same year, University services were 
rated significantly higher in areas of finance, registration and athletic services. Some of the 
rating differences between Web and paper-and-pencil groups which prevailed in 1997, 
diminished in 1999, particularly in areas pertaining to registration services, finances and 
places to study. This suggests that over time, services either changed at the University or those 
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A World Wide Web Response to Student Satisfaction Surveys 
Table 5. Ratings by Alumni Who Used the Web on Skills Learned at the University 



Graduate Survey 



Survey Item 


High 
n % 


1998 


Low 
n % 


Critical Thinking Skills 


481 8.9 




281 9.0 


Analytic Skills 


485 9.7 




275 8.7 


Creativity 


313 11.5 




448 7.8 


Ability to Learn 


452 9.7 




309 8.7 


Cooperate With Others 


428 9.6 




333 9.0 


Lead Groups 


324 10.5 




437 8.5 


Interaction w/ Diverse 








Groups 


405 9.4 




356 9.3 


Network w/ Others 


211 12.3 




549 8.0 


Job Skills 


300 10.7 




458 8.3 


Knowledge 


365 10.4 




397 8.1 


Computer Skills 


301 14.6 




458 5.9* 


Solve Problems 


459 9.8 




301 8.6 


Locate Info Materials 


294 12.2 




469 7.5* 


Adaptation to Change 


383 11.0 




373 7.8 


Persistence 


451 10.9 




312 7.1 



*P< .05 
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who responded via the Web became more like those who responded using paper-and-pencil 
formats, 



Table 6. Satisfaction Ratings In Other Areas of the Academic Experience by Students Who 
Responded To The Web Survey 





1997 


Student Experiences Survey 

1999 








High 


Low 


High 


Low 


Survey Item 


n % 


n % 


n 


% 


n 


% 


Cost 


357 5.0 


487 9.9* 


1338 


17.7 


1237 


12.3* 


Research 


608 7.9 


204 7.8 


1919 


15.9 


585 


12.3* 


Response to Concern 


335 4.5 


513 10.1* 


1380 


15.7 


1168 


14.2 


Cultural Diversity 


601 5.8 


250 12.8 


1981 


15.5 


583 


13.4 


Places to Study 


590 6.3 


266 11.3* 


1917 


14.9 


665 


15.5 


Motivation to Learn 


633 7.7 


233 7.7 


1907 


14.9 


694 


14.8 


Registration Services 


402 12.2 


446 3.6* 


1078 


16.2 


1429 


14.1 


Finance Services 


142 12.0 


431 6.0* 


538 


15.6 


1310 


14.8 


Athletic Services 


393 9.4 


234 3.8 * 


1177 


15.5 


696 


14.8 



*P< .05 
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Discussion 

This study was exploratory in nature and thus many of the trends and findings are 
intriguing, but inconclusive. In all cases, most students preferred to respond to the surveys 
through the paper-and-pencil format. 

Discussions with the persons who developed the Web format and findings from the 
literature suggest that responses to the Web-based format may have been higher had students 
received an email message containing a link to the Web page, if the Web format had included 
data encryption so that the students were assured of a higher level of confidentially in their 
responses, or if there were provisions for students to begin the on-line survey, leave it and 
return to it at a later time. 

Findings suggest that as students become more comfortable in using the Internet, their 
compliance in completing a Web-based survey will increase. Such trends were evidenced 
when the proportion of students who selected the Web survey doubled over a two-year period. 
They were further illustrated when younger students, who were more familiar with the 
Internet, selected a Web-based survey in higher proportions than upper classman or graduate 
students. 

Despite such findings, gender differences and some experiential variations prevail. This 
suggests that although Web surveys may serve as a window to the future, limiting oneself to 
one platform-particularly Web-in the current environment could bias the results obtained on 
certain items for particular student sub-populations, depending on the questions asked, ease in 
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A World Wide Web Response to Student Satisfaction Surveys 
accessing the form, length of the survey and/or way in which the survey appears on a 
particular student’s Web browser. Web-based surveys must therefore be well designed and 
easy for students to use, or students simply will not respond to them. More research is needed 
to facilitate the use of such modalities over the next three to five years. Following that, Web 
surveys may serve as the predominant format for surveys of students, faculty and staff. 
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